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(54) Method and system for field sequential color image capture 



(57) A method of field sequential color image cap- 
ture includes optically capturing a scene; filtering the 
scene through an active color filter (84) to product first 
cc-ior components, thereby modifying the spectral trans- 
mission of the scene as a function of time; detecting the 
scene with a single, monochrome sensor (90) and divid- 
ing the scene into multiple second color components, 
whereby each first color component of the scene is 
detected at a different point in time; aligning the second 
color components in time for each frame interval; stor- 
ing each color component in a memory unit; combining 
the stored second color components into a frame 
image; and processing the frame image for color repro- 



duction and format. A system for capturing a field 
sequential color image, includes an optical capture 
mechanism for capturing a scene frame-by-frame in 
frame intervals; an active color filer (84) for producing 
multiple first color components; a single monochrome 
area sensor (90); an array field selector (92) for dividing 
the scene into multiple second color components; multi- 
ple memory locations for storing each color component; 
a field-to-frame combiner (118) for combining the stored 
second color components into a frame image; and a 
color reproduction processor (120) for processing the 
frame image for color reproduction and format. 
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Description 

Field of the Invention 

[0001] This invention relates to the capture of color 5 
motion imagery by using sequential frame samples of 
different color filtration, referred to as field sequential 
color capture. 

background of the Invention ™ 

[0002] Color lag is the result of capturing color com- 
ponents of an image at different times during exposure 
of a single frame. One way to solve this problem is to 
use multiple sensors, each capturing a color compo- is 
nent. As will be explained, this approach has its own set 
of problems. There are, in the field of visual system 
modelling, three basic visual properties relating the 
luminance channel, as represented by video Y, and the 
opponent color channels, as approximated by the color 20 
difference signals, U and V. where B-Y=U and R-Y=V . 
These are: 

1. The maximum temporal frequency response of 
the opponent color system is less than 1/2 that of 25 
the luminance. 

2. The maximum spatial frequency response of the 
opponent color system is near 1/2 that of the lumi- 
nance. 

3. The maximum sensitivity of opponent color sys- 30 
tern is slightly greater than 1/2 that of the lumi- 
nance. 

The first and second properties are best described in A. 
B. Watson, Perceptual-components architecture for dig- 35 
ital video, JOSA A V. 7 # 1 0, pp. 1 943-1 954, 1 990; while 
the third property is described in K. T. Mullen The con- 
trast sensitivity of human colour vision to red-green and 
blue-yellow chromatic gratings J. Physiol. V. 359, pp. 
381-400, 1985. These three properties will be used in 40 
the selection of the relative durations of exposures in 
order to prevent color lag and in the spatio-temporal 
integration following image capture. 
[0003] L Arend et at, Color breakup in sequentially 
scanned LCDs. SID 94 Digest, pp 201-204, 1994; and 45 
D. Post et ai Predicting color breakup on field-sequen- 
tial displays, SPIE Proc. V. 3058, pp 57-65, 1997, spe- 
cifically investigate color break-up, or color lag. 
Although the research behind these papers is intended 
for application to field sequential color displays, some of so 
the findings are relevant to color field sequential capture 
(FSC). One of these is that color lag detection has a 
strong luminance component. In fact, as the color offset 
between R, G and B layers is increased from zero to 
detectability, the first signal to exceed threshold is the 55 
luminance. This manifests itself as blurring in texture 
before any high contrast edge blur occurs, because of 
the masking effects caused by high contrast edges. 



Eventually, as the offset is increased, color artifacts 
become visible at high contrast edges. 
[0004] Hunt, The reproduction of colour in photog- 
raphy printing, and television, 4th edition, pp. 409-410, 
Fountain Press, England, describes work done with 
YRB camera systems. These were primarily investi- 
gated for 3-sensor systems when the sensors where 
tubes. The YRB system attempted to reduce the visibil- 
ity of the color offset problem due to manufacturing tol- 
erances by optically aligning the tubes. Due to 
complications with gamma correction, it was aban- 
doned in favor of a RGBY systems using four tubes. 
These complications, which were extremely difficult to 
resolve in analog systems, are now easily resolved in 
digital signal processing. 

[0005] In order to appreciate the advantages of field 
sequential color capture, two other common 
approaches must be understood. One other approach is 
the use of three sensors, typically using red, green, and 
blue filtration, which simultaneously capture the scene's 
dynamic content. This technique is used with both tube 
pickup devices and with 2D sensor arrays, such as 
charge coupled devices (CCD) or composite metal-on- 
silicon (CMOS) devices, which are referred to in the lit- 
erature as 3-CCD cameras. 

[0006] Another approach is the use of a single 2D 
sensor having color filtration applied separately to each 
pixel. The colors are arranged in spatially varying man- 
ners designed to provide a high spatial bandwidth for 
luminance or green, and to minimize color aliasing arti- 
facts. The result is that each color layer has incomplete 
samples per frame, however, special interpolation algo- 
rithms are used to reconstruct full dimensioned frames 
for each color layer. This approach is known as color fil- 
ter array (CFA) camera capture. 

[0007] The 3-CCD, or 3-tube, approach's chief dis- 
advantage is the cost of three sensors. A second disad- 
vantage is the problem of color mis-registration between 
the three sensors due to their alignment in the optical 
path relative to the scene, which may impose tight man- 
ufacturing tolerances that increase manufacturing 
costs. Color mis-registration may cause luminance blur 
in textures with very small amounts of mis-registration, 
and may cause color bleeding, also referred to as color 
fringing, at both achromatic and chromatic edges. If the 
registration is well aligned, the 3-CCD approach 
achieves the resolution of the sensors for all three color 
layers of a frame. Due to cost issues and manufacturing 
constraints, however, these approaches are only used 
for high-end studio video cameras, and digital still cam- 
eras aimed for the professional and advanced hobbyist. 
[0008] The CFA approach is less costly because it 
uses only a single sensor, however, its disadvantages 
include reduced spatial resolution, the necessity for an 
interpolation algorithm to reconstruct the three color 
frames for display, and necessity for an anti-aliasing fil- 
ter to prevent diagonal luminance high spatial frequen- 
cies from aliasing into lower frequency color patterns 
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and to prevent color high spatial frequencies from alias- 
ing into luminance or color patterns. Consequently, 
there is a trade-off between sharpness and color arti- 
facts. These artifacts are quite noticeable in such com- 
mon image content as highlight reflections in eyes, and 5 
the expected luminance high spatial frequencies in tex- 
ture, e.g., hair, or geometric patterns. In current imple- 
mentations, fairly complex interpolation algorithms, 
which include pattern recognition, are used to try to 
maximize sharpness and minimize color spatial arti- 10 
facts. The sharpness/aliasing tradeoff may be 
described via Fig. 3, using either filter 42 or filter 44. 
Either fitter may be increased in bandwidth by scaling 
their shape on the frequency axis. Though the image 
will appear to be sharper, signals having a frequency 75 
higher than that of the digital Nyquist will be aliased, 
i.e., the signal will fold over to frequencies lower than 
that of the digital Nyquist. These false lower frequencies 
are, in effect, added to the true signal. If all of the alias- 
ing is removed, however, the image will appear blurred. 20 
It therefore requires a certain amount of craftsmanship 
in designing the filters to provide an appropriate amount 
of aliasing in order to maintain the bandwidths as high 
as possible. Most camera manufacturers opt to avoid 
any chromatic aliasing, because it is a new categorical 25 
distortion, and favor the sharpness reduction, which is 
already present to some extent In summary, the two 
common approaches do not achieve the resolution of 
their sensors dimensions, either in luminance or in 
color. 30 
[0009] The foregoing characteristics are depicted in 
Fig. 1-6. In Fig. 1, the Nyquist folding frequencies for the 
typical 2D CFA, also known as a Bayer pattern, are 
depicted, which are collectively referred to as the 
Nyquist boundary. Line 22, with a slope of -1, is the 35 
boundary for green. The green boundary may achieve 
the value of 0.5 cycles/pixel (cy/pix) only at the horizon- 
tal and vertical frequencies. The red and blue compo- 
nents are shown by Line 24 and are restricted to the 
square region limited by 0.25 cy/pix. Generally these 40 
color signals are manipulated so that G captures lumi- 
nance, and R and B capture chrominance difference 
signals, by taking their difference or ratio with G. This 
means that luminance information with higher frequen- 
cies than those indicated in triangles 26, 28, which will 45 
alias into lower frequencies, showing up as both lumi- 
nance and chrominance alias patterns. Similar effects 
occur for chrominance information outside of the 
smaller Nyquist square 24 boundary. Because it is most 
critical to get minimally aliased luminance information in so 
an image, only the luminance Nyquist boundaries in the 
figures are discussed herein. 

[0010] Fig. 2 includes several additional elements. 
Fig. 2a is a similar representation depicted in 3D. Con- 
tours 30 (Fig. 2 only) depict the human visual system ss 
contrast sensitivity function (CSF), which is analogous 
to its frequency response. This CSF is mapped to the 
digital frequency plane, based on calibration via the size 



of the image in pixel heights, and the viewing distance 
as expressed in units of picture heights. This mapping is 
a way of normalizing the actual physical display size 
and viewing distance. In Fig. 2, the displayed image size 
is 480 pixels high and the viewing distance is 6 picture 
heights (6H), which is the traditional NTSC image size. 
The CSF is a 2D surface, whose values are shown as a 
contour plot, with a peak frequency response near .08 
cy/pix, for this viewing distance, which is six picture 
heights (H). The actual distance is dependent on the 
size of the display, however, "normalization" eliminates 
the need to include the actual viewing distance. The 
highest frequency contour 31 corresponds to the high- 
est frequency the human eye can see. 
[0011] The Nyquist boundary 32 for a full frame 
image capture system is depicted, which is the case for 
either 3-CCD systems or FSC color systems. This 
region is bounded by 0.5 cy/pix in both the horizontal 
and vertical directions, also referred to as "shutter 
RGB". Area 34, located between the triangle formed by 
the luminance CFA boundary 22 and rectangle 32, is 
the resolution advantage of the full-frame over the CFA 
approaches. While this looks impressive, resulting in a 
doubling of the area of the CFA boundary, it is tempered 
by the visual system frequency response, whose anisot- 
ropy causes its isocontours to more closely follow the 
CFA pattern. 

[0012] The final addition to Fig. 2 is the optical 
antialiasing filter which is used to prevent frequencies 
higher than the CFA Nyquist limit from reaching the sen- 
sor. If we assume the system's focusing lens is used for 
invoking this function, it is generally isotropic, and to 
prevent aliasing at 45 degrees, we must select its band- 
width as shown by circular line 36. Note that some high 
vertical and horizontal frequencies that are actually 
Lower than Nyquist limits imposed by the CFA are lost, 
as represented by area 38, located between CFA 
boundary 22 and optical cutoff filter boundary 26. 
[0013] Lens optics are not generally used for 
antialiasing. A birefringent crystal filter is used because 
it has a better frequency response for a given cut-off fre- 
quency, as shown in Fig. 3, generally at 40. The optical 
transfer function (OTF) is the frequency response of the 
lens in use. In Fig. 3, the frequency response of an opti- 
cal lens may at best be diffraction limited, and that 
response is shown as curve 42, which is designed to 
cut-off at the Nyquist of 0.5 cy/pix. The birefringent filter 
merely causes a displacement in the beam, resulting in 
two impulse responses in the spatial domain, which 
may be modeled as a cosine in the frequency domain. 
This response is shown as line 44, which is designed for 
the same cutoff frequency of 0.5 cy/pix. 
[0014] Although the birefringent crystal filter has 
better frequency response characteristics for a given 
cut-off frequency, the usual technique is to have one 
crystal displace a pixel horizontally, and to then cascade 
another crystal which causes a vertical displacement, 
as may be seen in Fig. 3. The net effect is a Cartesian- 
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separable low-pass filter (LPF) which is indicated in Fig. 
4, generally at 50, where the cut-off frequencies are 
selected to avoid any luminance aliasing at the critical 
point at 45 degrees. Consequently, the horizontal and 
vertical Nyquist boundaries 52 end up at only 0.25 
cy/pix, which strongly reduce the resolution potential of 
the sensor, thus reducing sharpness. Since various 
design aims are used to trade-off aliasing vs. sharp- 
ness, the typical CFA camera system frequency 
response will lie between square 52 and line 22, in two 
triangular regions 54. One may see the substantial 
increase in resolution area afforded by a full sensor 
frame capture system depicted by rectangle 56 inter- 
secting at 0.5 cy/pix in the horizontal and vertical. 
[0015] Much of the work in the FSC field was done 
prior to the NTSC color standard, when color field 
sequential was a competitor for color television. As a 
consequence, most of the prior art references are old. 
Analog field sequential color video was difficult to 
achieve at high frame rates and found its key applica- 
tions in telecine and other specialized applications. Only 
recently has the activity picked up due to full digitization 
of the video system, which makes it easier to couple 
field sequential color capture to simultaneous color dis- 
plays. 

[0016] U.S. Patent No. 2,738,377, to Weighton, 
granted March 13, 1956 for Color Television, describes 
a color television system which uses a rotating color fil- 
ter wheel, with equally spaced wedges in the order 
GRGBGRGB, with a single pickup tube, and a CRT to 
constitute a full color television system. The reference 
describes an eight-fold interlace, with the color wheel 
spinning fast enough to cause a different color filter for 
each interlace line. A sufficient number of lines are used 
to result in a captured image with 400 G Lines and 200 
R and B lines. The main difficulty with this approach is 
the extreme demand imposed on the system bandwidth 
due to the eightfold interlace. Another is that color field 
sequential display is required in this system, and 
because of eye movements, such displays are more 
susceptible to color breakup than color field sequential 
capture. 

[0017] U.S. Patent No. 3,604,839 to Kitsopoulos, 
granted September 14, 1971 for Field-sequential color 
television apparatus employing color filter wheel and 
two camera tubes, describes a television system having 
a color filter wheel and two camera tubes, which is pri- 
marily aimed at telecine. The purpose of the two tubes 
is to allow for simultaneous capture of two different 
colors, thus allowing the exposure to lengthen, given the 
field rate constraints, and increase the signal to noise 
ratio. 

[0018] U.S. Patent No. 3,969,763 to Tan, granted 
July 13, 1976, for Color television camera provided with 
a pickup device and color filter placed in front thereof, 
describes a color field sequential camera that uses a 
single pickup tube and a color filter wheel with many fine 
color strips. The purpose is to capture the color filtered 



sections of an image more spatially and temporally 
coincident. It approaches a color sequential interlace, 
and is primarily an analog hardware system addressing 
various delay and convener steps. The reference is 

5 notable because it discloses a system which uses liquid 
filters that are electronically controllable, rather than 
using a mechanical wheel. The system also captures in 
a YRB space, rather than the more common RGB. 
[0019] U.S. Patent No. 4,067,043, to Perry, granted 

w January 3, 1978 for Optical conversion method, 
describes the use of electro-optical ly controlling color 
filtration, in a field sequential mode, via the use of 
crossed polarizers. 

[0020] U.S. Parent No. 4,851 ,899 to Yoshida et al, 

is granted July 25, 1 989, for Field-sequential color televi- 
sion camera, describes an RGB field sequential camera 
using a color wheel that causes the field order to be 
R1R2G1G2B1B2, etc. Because the first field of each 
color pair has a residual charge from the previous, drf- 

20 ferent color, the system discards the first captured field 
in order to prevent color desaturation. 
[0021] U.S. Patent No, 4,967,264 to Parulski et al., 
granted October 30, 1 990, for Color sequential optical 
offset image sampling system, combines color field 

25 sequential concepts with sensor dithering to increase 
resolution. A color filter wheel is used containing, in 
order, 2 green, one red, and one blue element. The two 
green elements are placed at various tilts to create spa- 
tial offsets on the sensor, so the two green fields may be 

30 combined to create an image with higher green resolu- 
tion than would be available solely from the sensor. 
[0022] U.S. Patent No. 5,084,761 to Nitta, granted 
January 28, 1 992, for Video camera with rotatable color 
filter and registration correction, addresses the color 

35 mis-registration caused by the variations in tilt and thick- 
ness of the color filter wheel in color field sequential 
cameras. The horizontal and vertical deflection signals 
are adjusted. Even though the color field sequential 
camera is, by design, nor supposed to have a color reg- 

40 istration problem, it does in fact have this problem 
because of the mechanics of the color wheel. The refer- 
ence seeks to avoid this problem with electronically con- 
trolled color fitters, such as a LCD. 
[0023] U.S. Patent No. 5,548,333 to Shibazaki etaf, 

45 granted August 20, 1996, for Color mixing prevention 
and color balance setting device and method for afield- 
sequential color television camera, addresses color 
mixing and color balance processes in a RGB filter 
wheel field sequential color camera. It allows a video 

so field with a mixed color input, that is a transition from 
one color sector to another of the color wheel, to be 
used, but discards the signal during the mixed color 
time interval. 

[0024] Another goal of the system is to adjust timing 
55 intervals to achieve white balance, also referred to as 
color automatic gain control (AGC). The timing intervals 
are determined from calibration with a white card to 
compensate for studio lighting color temperature. Due 
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to the nature of the color wheel, the RGB exposure 
durations retain the same color lag, even if the exposure 
durations are shortened. This is illustrated up in Fig. 7, 
where the issues of discarded fields and color mixing 
have been simplified to highlight the exposure control, 
which is used for color temperature matching. Top row 
56 depicts the exposure durations at a maximum per 
frame, with the horizontal axis being time, for a total of 
three frames. Bottom row 58 depicts a reduction in tem- 
poral exposure, where white bars indicate periods of no 
exposure, or non-capturing time, to the sensor accumu- 
lator. The start time of each color field is fixed in time 
due to the fixed patterns on the physical color wheel, so 
even though the durations are shortened, the color lag, 
as measured from either center-to-center positions, or 
from like edge positions, remains the same. A gap 59 is 
depicted as the non-capturing time between color com- 
ponents. Such a gap will be understood to also be 
present between each frame. 

[0025] U.S. Patent No. 5,748,236, to Shibazaki, 
granted May 5, 1998, for Color mixing prevention and 
color balance setting device and method for a field- 
sequential color television camera, is based on a con- 
tinuation-in-part application from the '333 patent, supra, 
however, this reference concentrates on the hardware, 
and specifically, the concept of discarding the accumu- 
lated charge during the mixed interval. 
[0026] U.S. Patent No. 5,751 ,384 to Sharp, granted 
May 1 2, 1 998, for Color polarizers for polarizing an addi- 
tive color spectrum along a first axis and its compliment 
along a second axis, describes switchable color filter via 
polarization and LCD as would be used in digital still 
and video cameras. 

[0027] U.S. Patent No. 5,767,899 to Hieda et a/., 
granted June 16, 1998, for Image pickup device, 
addresses the problem of constant luminance, where 
the luminance signal is formed prior to any gamma cor- 
rection. This is opposed to conventional video process- 
ing, where the gamma correction nonlinearity is applied 
to the RGB values prior to the formation of the lumi- 
nance signal. In this device the captured Y, Cr, and Cb 
outputs from a CFA interpolation process are converted 
to RGB, gamma corrected, then converted to conven- 
tional YR-YB-Y. Various base-clipping, or coring, limit- 
ing, and color suppression based on the luminance 
signal are then incorporated. 

Summary of the Invention 



color components in time for each frame interval; stor- 
ing each color component in a memory unit; combining 
the stored second color components into a frame 
image; and processing the frame image for color repro- 

5 duction and format. 

[0029] A system for capturing a field sequential 
color image, includes an optical capture mechanism for 
capturing a scene frame-by-frame in frame intervals; an 
active color fitter for producing multiple first color com- 

w ponents; a single monochrome area sensor; an array 
field selector for dividing the scene into multiple second 
color components; multiple memory locations for storing 
each color component; a field-to-frame combiner for 
combining the stored second color components into a 

15 frame image; and a color reproduction processor for 
processing the frame image for color reproduction and 
format. 

[0030] An object of this invention is to provide a 
color image capture system and method using only a 
20 single sensor while capturing the full resolution of the 
sensor for all three color planes. 

[0031] Another object is to provide a system and 
method wherein color lag will decrease as shutter dura- 
tions decrease. 

25 [0032] A further object of the invention is to provide 
a system and method that reduce color mis- registration. 
[0033] These and other objects and advantages of 
the invention will become more fully apparent as the 
description which follows is read in conjunction with the 

30 drawings. 

Brief Description of the Drawings 



35 



[0034] 



Fig. 1 depicts the frequency capability of a color fil- 
ter array (CFA) sensor. 

Fig. 2 compares CFA and full frame sensors, iso- 
40 tropic optics, and the human visual system. 

Fig. 2a is a 3D representation of Fig. 2. 

Fig. 3 depicts frequency response of drffraction-lim- 
45 ited optics and biref ringent crystal. 

Fig. 4 depicts CFA and full frame sensors, 2D bire- 
fringent optics and human visual system. 



[0028] A method of field sequential color image so 
capture includes optically capturing a scene; filtering 
the scene through an active color filter to produce first 
color components, thereby modifying the spectral trans- 
mission of the scene as a function of time; detecting the 
scene with a single, monochrome sensor and dividing 55 
the scene into multiple second color components, 
whereby each first color component of the scene is 
detected at a different point in time; aligning the second 



Fig. 5 depicts 4H VD at standard resolution. 

Fig. 6 depicts 4H VD at high resolution. 

Fig. 7 depicts the effect of reducing shutter dura- 
tions in prior art field sequential color. 

Fig. 8 is a block diagram of the system of the inven- 
tion. 
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Rg. 9 depicts color field timing. 

Fig. 1 0 depicts finely dithered color field timing. 

Rg. 1 1 depicts temporal subsampling implementa- 
tion of color field timing. 

Rg. 12 depicts a specific implementation of an 
array field selector. 

Rg. 13 depicts an alternate implementation of an 
array field selector. 

Detailed Description of the Preferred Embodiment 

[0035] The field sequential color approach of this 
invention attempts to achieve the primary advantages of 
the 3-CCD approach and the CFA approach, while 
removing some of their disadvantages. Theoretically, 
the invention may achieve the full frame resolution of the 
3-CCD approach for all color layers of a frame. Because 
there is only one sensor, the color mis- registration prob- 
lem due to mechanical alignment is eliminated. The only 
static color mis-registration in the invention is due to the 
optics, generally corrected by achromatizing lenses. It 
also accomplishes full frame resolution without 
mechanical color mis-registration with one mono- 
chrome sensor, exceeding the cost advantages of the 
CFA approach. Further it does not require the use of 
interpolation algorithms. 

[0036] The system of the invention does have its 
own particular problems, however. The most predomi- 
nant is the result of capturing the color fields at separate 
points in time, which results in a misalignment of the 
colors of a moving edge when the color fields are com- 
bined into a single full-color frame. The moving edge 
may be the result of either object motion or camera 
motion. This effect is known as "color breakup" or "color 
lag." This effect may be analyzed as a localized or 
motion -dependent color mis-registration, causing both 
luminance blur and color bleeding of moving edges. In 
some cases, such as still scenes, this color mis-regis- 
tration is much less than the color mis-registration of the 
3-CCD system, but in other cases, such as with fast 
motion, the color mis-registration may be greater than 
with the typical 3-CCD approach. 

[0037] Another problem is in achieving three sepa- 
rate color captures in the duration of a single video 
frame, which is on the order of 1 7-33 ms. This was tra- 
ditionally accomplished by a rotating color filter wheel, 
which required expensive mechanical engineering. Fur- 
ther, having a component rotating at such high veloci- 
ties made the cameras more susceptible to mechanical 
failure, requiring frequent repair and re-calibration. 
Recently, however, a new method to sequentially modu- 
late color has been developed using electrically-control- 
lable LCDs. These LCDs switch colon through a single 
optical pathway fast enough to accomplish more than 



three switches during a video frame, and have the addi- 
tional advantage that there are no moving parts. The 
problem of color breakup is reduced by the method of 
the invention. 

5 [0038] The various spatial frequency responses of 
these three systems, /.e., CFA, 3-CCD, and FSC, are 
shown in Figs 5 and 6, plotted against the visual sys- 
tems 2D contrast sensitivity function (CSF) for more 
viewing distances and displayed image resolutions. Rg. 

w 5 depicts, generally at 60, the CSF for a standard defini- 
tion (SD) resolution of 480 lines, but with a more 
demanding viewing distance of four picture heights 
(4H). The full frame approaches for 3-CCD and color 
field sequential, have the same spatial frequency 

is response, bounded by rectangle 62 at 0.5 cy/pix. The 
CFA has the smaller range, as indicated by the triangle 
under line 64. 

[0039] In Fig. 6, the CSF is mapped, generally at 
66, to a digital frequency plane for a high definition 

20 image and of 960 lines and a viewing distance of 4 pic- 
ture heights, which is appropriate for high definition 
viewing. It may be seen that as the viewing distance 
increases or the number of display lines increases, the 
CFA begins to match the maximum frequency capabili- 

25 ties of the visual system. Thus the CFA will generally be 
more optimum for far viewing distances, or high pixel 
resolution image display, than the full frame 
approaches. With more demanding viewing distances, 
i.e., closer, the full-frame approaches will have higher 

30 perceived resolution. 

[0040] The sensor of the invention is monochrome, 
as opposed to that used in the CFA approach. The sen- 
sors used in CFA systems have increased manufactur- 
ing costs due to the requirement that an individual filter 

35 be placed on each pixel. The reduction from three sen- 
sors to one sensor is done so that color breakup is 
reduced below visibility, and further, so that no perceiv- 
able increases in noise occur, despite the shorter cap- 
ture periods needed per color to maintain the same 

40 frame rate as with a 3-CCD or CFA camera system. To 
this end, images are captured as close as possible in 
the YUV video space, where Y is luminance, U is a 
scaled B-Y color difference and V is a scaled R-Y color 
difference. The key aspect of this capture is to capture Y 

45 directly, so that color offsets due to camera motion or 
object motion do not have any effect on the Y signal. 
One preferred embodiment of the invention achieves 
this full-resolution full-color system with only a doubling 
of the frame rate, as opposed to a tripling. One objective 

so is to adjust the shutter durations, either for motion rendi- 
tion, or for color balance purposes, so that as the shut- 
ter duration decreases, the color lag will decrease as 
well. 

[0041] The method relies on key characteristics of 
55 the human visual system (HVS) relating the luminance- 
to-opponent color channels. The ratios of opponent 
color to luminance are approximately 1/2 for temporal 
bandwidth, spatial bandwidth, and maximum sensitivity. 
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These ratios are used in the selection of the relative 
durations of exposures and in the spatio-temporal inte- 
gration following image capture. The duration and con- 
figuration of the exposures is designed to prevent color 
lag. The spatio-temporal signal processing following 
capture is designed to minimize the perception of 
increased noise, which results from allocating less 
exposure duration to both the color difference and lumi- 
nance signals relative to a 3-CCD or CFA image cap- 
ture. The temporal bandwidth ratio of color-to- 
luminance indicates that one may temporally low-pass 
filter or temporally sub-sample the color difference sig- 
nals relative to the luminance. Temporally averaging of 
samples after the A/D may be used to reduce the noise 
caused by shorter exposure durations afforded to the 
color difference signals. The spatial bandwidth ratio of 
color-to-luminance allows for spatial low-pass filtering of 
the captured color difference signals which may also 
reduce the noise caused by shorter exposures. Both of 
these effects may translate into shorter exposures for 
RG and BY than Y. Finally the ratio of sensitivity of 
color-to-luminance means that the color system may 
have 2" 1 ^ the noise of the luminance and yet have the 
same visibility. 

[0042] Referring now to Fig. 8, the invention is 
depicted generally at 80. A scene 82 is captured via 
optics and passes through an active color filter 84, 
resulting in a first color component set, which includes 
B, Y and R first color components, which changes the 
spectral transmission of the image as a function of time, 
and which is controlled by a color field control clock 86, 
which, in turn is controlled by the frame control clock 88, 
which operates at a frame rate of 60 frames per second 
(fps) for current progressive formats, but which may be 
set to any desired and achievable rate. The filtered 
scene impinges a monochrome area sensor 90, which 
nominally detects and captures three color fields. These 
three images are clocked to an array field selector 92 at 
the field rate, which is approximately three times faster 
than the frame rate. However, the color field clock 
pulses non-uniformly during sampling of the frame inter- 
val, as will be described later herein. The three fields 
captured in different points in time are distributed to the 
U (94), Y (96) and V (98) channels, also referred to 
herein as second color components forming a second 
color component set, and are aligned in time in the 
same frame interval in the array field selector. The U, Y 
and V fields are stored in memory as three spatial 
images per frame, 100, 102 and 104, respectively. The 
two color difference images, U and V, are then filtered 
by first low-pass temporal filters 106, 108, and then by 
low-pass spatial filters 110, 122, respectively, to reduce 
the noise due to their shorter exposure intervals. The 
luminance image, Y, may optionally also be filtered by a 
low-pass temporal filter 1 1 4 and/or by a low-pass spatial 
filter 116. Alternately, bandpass filtering may be used 
instead of low-pass filtering. The three separate field 
images are then combined into a frame image by a field- 



to-frame combiner 118, and processed according to 
protocols of color reproduction and format by a color 
reproduction processor 120, for eventual coding for 
transmission to recorders and other devices as a sys- 

5 tern color metric 1 22. 

[0043] A basic feature of the invention is shown in 
Fig. 9, generally at 124. This is the nonuniform temporal 
sampling of the three color planes per frame, which may 
be compared to Fig. 7. In Fig. 7, the trigger points for 

w each color component are the same and the color lag is 
the same regardless of the shutter speed. Non-captur- 
ing time 59 is dispersed throughout the frame. The 
majority of the available frame capture time is used to 
sample that part of the spectrum, which contributes to 

15 luminance component. This value is nominally 75%, but 
the area sensor clocking rates may force other values to 
be used. The top row 126 of Fig. 9 depicts the color field 
capture intervals for the maximum possible duration 
within a frame. Again, a slight non-capturing time gap 

20 127 exists between the capture of each color compo- 
nent within a frame, and also between frames. The color 
fields, or components, are indicated U, Y, V, even though 
it is not possible with current technology to optically cap- 
ture the U and V signals. However, the terms U and V 

25 are used to indicate the color filter fields that will lead 
predominately to the U, Y and V signals. The trigger 
points are moved in row 128 as the shutter speed is 
increased, which results in a decrease in color lag or off- 
set. 

30 [0044] In most cases, these filters will be B, Y, and 
R. The UYV order is preferred and was arrived at 
through experimentation. This order minimizes the color 
offset of the color differences from luminance, as there 
is some minimal luminance leaking into the U and V sig- 

35 nals. The bottom row 128 of Fig. 9 depicts the system 
behavior as the "shutter speed" is increased, /.e., the 
total exposure duration is shortened in order to enable 
better motion rendition, reduce the aperture time if the 
light level is very high, or for color balance functions, in 

40 which case the UYV intervals will be changed relative to 
each other. Color lag 129 may best be characterized by 
the center time between the different YUV fields of a 
single frame. The color lag for row 128 is markedly less 
than that for row 126 with its slower shutter speed. In 

45 row 1 28, capturing takes place in the temporal center of 
each frame. Capturing the color components includes 
capturing each color component of the first color com- 
ponent set in a time interval that is less than the inverse 
of the number of color components in the first color 

so component set. This is an advantage of the invention 
because the color offset is reduced as the shutter speed 
is increased. The behavior of the prior art in Fig. 7 suf- 
fers from the same color offset regardless of shutter 
durations, as seen from the color log markers. It will be 

55 appreciated tat "capturing* actually takes place over the 
entire duration of the frame during the open exposure 
time of each frame, however, the center-of-mass of the 
exposure is in the temporal center of each frame. 
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[0045] In the most common applications, the inven- 
tion requires a color field rate of three times the frame 
rate, which in turn requires a 90 fps data rate for the 
transfer from imager to memory for a system using a 
frame rate of 30 fps. This is applicable to interlace at 60 5 
interlace fields/sec, or a progressive system at 30 fps. 
For a progressive system at 60 fps, the data unloading 
from the imager works at a rate of 1 80 fps. 
[0046] In a modified form of the invention, the UYV 
samples are more finely sub-divided within a frame, as 10 
shown in Fig. 10, generally at 130. Top row 132 depicts 
the color field capture intervals for the maximum possi- 
ble duration within a frame. Bottom row 134 depicts the 
behavior for increased shutter speed. More finely dither- 
ing the UYV samples within a frame reduces the color 15 
breakup artifact, but forces the data race off the sensor 
to be faster. The rate of the active color filter also needs 
to be similarly increased, but currently the limiting tech- 
nology is in getting the data off the imager. In Fig. 10, 
two color reproduction units (CRUs) are depicted, which 20 
form the UYV temporal triad, for each frame. More 
CRUs may be used, at the cost of more expensive hard- 
ware. This technique reduces the color breakup, trans- 
forming it more into a familiar luminance blur that is no 
greater than that observed for the equivalent shutter 25 
speed for CFA or 3-CCD cameras. As in Fig. 9, bottom 
row 134 depicts the behavior for faster shutter speeds, 
where once again the color offset 135 reduces with 
decreasing shutter durations. 

[0047] The results of using an embodiment of the 30 
invention that imposes the lowest burden on the imager 
data transfer is depicted in Fig. 1 1 , generally at 136. Top 
row 138 depicts the color field capture intervals for the 
maximum possible duration within a frame. Bottom row 
140 depicts the behavior for increased shutter speed. 35 
This embodiment of the invention only requires a data 
rate twice that of the frame rate. In each frame a Y sam- 
ple is taken, but the U and V components are sampled 
only in alternate frames. This results in an effective tem- 
poral subsampling of the color difference signals by a 40 
factor of 2, which is known to be visually equivalent to 
full temporal sampling, due to the visual properties pre- 
viously identified herein. In fact, even though the two 
color difference samples are 180 degrees out of phase, 
this result is indistinguishable from the case where the 45 
color difference images are sub-sampled by 2, but kept 
in phase. This system may be achieved using a modest 
60 fps chip, resulting in 30 fps Y and 15 fps R-Y and B- 
Y progressive signals. As in Figs. 9 and 10, bottom row 
1 40 depicts the advantageous behavior of the invention so 
for faster shutter speeds. 

[0048] The next two embodiments of the invention 
impact the design of the array field selector of the inven- 
tion. Fig. 1 2 depicts the first of these embodiments of an 
array field selector 150 wherein the active color filters ss 
1 52 are B, Y and R. Frame control 88 is used to gate the 
three color fields to a single frame interval, while color 
field control clock 86 directs the B, Y and R images 



through a switch 154 to separate arrays 156, 158, 160, 
respectively, in memory. A B-Y differentiator 1 62 and a 
R-Y differentiator 164 result in the formation of the U 
and V color difference signals from these captured 
wavelength spectra. The specific scaling and offsets to 
make the U and V signals compatible is applied in the 
color reproduction processing mechanism/step of Fig. 
8. 

[0049] The next embodiment is depicted in Fig. 13, 
where the U and V signals are generated in the analog 
domain for an array field selector 1 70 with temporal dif- 
ferentiators 1 72, 1 74 applied across the U and Y transi- 
tion, and the Y and V transition, respectively. The A/D 
conversions 1 76, 1 78, 1 80, occur after the analog differ- 
entiation, and the images are held throughout the dura- 
tion of the frame. This embodiment has the advantage 
of reducing bit-depth noise, and consequently, bit-depth 
requirements, for the color difference signals. Depend- 
ing on the temporal characteristic of the differentiators, 
this may also concentrate the signal energy going into 
the U and V channels to be more temporally contiguous 
to the Y channel. 

[0050] Although the Y filter characteristics may be 
white or green, other embodiments using any lumi- 
nance function, such as V x , may be used, so long as the 
filter characteristic is as close to the V-function, which 
appears as yellow-orange color, as possible. 
[0051] Although a preferred embodiment, and sev- 
eral variations thereof have been described, it will be 
appreciated by those of skill in the art that further varia- 
tions and modifications may be made thereto without 
departing from the scope of the invention as defined in 
the appended claims. 

Claims 

1. A method of field sequential color image capture 
comprising: 

optically capturing a scene frame-by-frame; 

filtering the scene through an active color filter 
to product a first color component set having 
plural first color components therein, thereby 
modifying the spectral transmission of the 
scene as a function of time; 

detecting the scene with a single, monochrome 
sensor, wherein each first color component of 
the first color component set of the scene is 
detected at a different point in time within a 
frame; 

dividing the detected scene into a second color 
component set having plural second color com- 
ponents therein; 

aligning the second color component set in 
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time for each frame interval; 

storing each second color component in a 
memory unit; 

combining the stored second color compo- 
nents into a frame image; and 

processing the frame image for color reproduc- 
tion and format 

2. The method of claim 1 wherein said dividing 
includes dividing the scene into U, Y and V color 
components. 

3. The method of claim 2 which includes, after said 
storing, filtering at least some of the second color 
components. 

4. The method of claim 3 wherein said filtering 
includes temporally and spatially filtering the U and 
V color components. 

5. The method of claim 3 wherein said filtering 
includes temporally and spatially filtering the Y 
color component 

6. The method of claim 1 wherein said modifying 
includes operating a color field clock at a first pre- 
determined rate and operating a frame control clock 
at a second predetermined rate, wherein the first 
predetermined rate is greater than the second pre- 
determined rate. 

7. The method of claim 6 wherein said operating 
includes setting the first predetermined rate to 
about 3X that of the second predetermined rate. 

8. The method of claim 6 wherein said operating 
includes setting the first predetermined rate to 
about 2X that of the second predetermined rate. 



w 
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The method of claim 1 wherein said filtering 
includes filtering the scene to capture B, Y and R 
first color components. 45 



10. The method of claim 9 wherein said dividing 
includes storing B, Y and R first color components 
and differentiating B - Y to form a U second color 
component and differentiating R - Y to form a V 
color component 



50 



11. The method of claim 10 wherein said differentiating 
is temporal differentiating. 

12. The method of claim 1 wherein said capturing 
includes directly capturing a luminance color com- 
ponent. 



55 



13. The method of claim 1 wherein said capturing 
includes capturing each first color component of the 
first color component set in a time interval that is 
less than the inverse of the number of first color 
components in the first color component set. 

14. A system (80) for capturing a field sequential color 
image, comprising: 

an optical capture mechanism for capturing a 
scene frame-by-frame in frame intervals which 
operates at a predetermined shutter speed; 

an active color filter (84) for producing a first 
color component set having plural first color 
components therein, thereby modifying the 
spectral transmission of the scene as a func- 
tion of time; 

a single monochrome area sensor (90) for 
sensing said first color components, wherein 
each first color component of the scene is 
detected at a different point in time within a 
frame; 

an array field selector (92) for dividing the 
scene into a second color component set hav- 
ing plural second color components therein; 

a color field control clock (86) for controlling 
said active color filter (84), said area sensor 
(90) and said array field selector (92) at a first 
predetermined rate and for aligning the second 
color components in time for each frame inter- 
val; 

multiple memory locations for storing each 
color component; 

a field-to-frame combiner (118) for combining 
the stored second color components into a 
frame image; 

a color reproduction processor (120) for 
processing the frame image for color reproduc- 
tion and format; and 

a frame clock (82) for controlling said colorfield 
clock (86), said memory locations, and said 
field-to-frame combiner (118) at a second pre- 
determined rate, wherein said first predeter- 
mined rate is greater than said second 
predetermined rate. 

15. The system (80) of claim 14 wherein said arrayfield 
selector (92) divides the scene into U, Y and V color 
components. 
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16. The system (80) of claim 15 which includes tempo- 
ral and spatial filters (106, 108 and 110, 112) for 
temporally and spatially filtering at least some of the 
second color components. 

5 

17. The system (80) of claim 1 6 which includes tempo- 
ral and spatial filters (106, 108 and 110, 112) for 
temporally and spatially filtering the U and V color 
components. 

10 

18. The system (80) of claim 16 which includes tempo- 
ral and spatial filters (106, 108 and 110, 112) for 
temporally and spatially filtering the Y color compo- 
nent 

15 

19. The system (80) of claim 14 wherein said first pre- 
determined rate is 3X that of the said second pre- 
determined rate. 

20. The system (80) of claim 14 wherein said first pre- 20 
determined rate is 2X that of the said second pre- 
determined rate. 

21. The system (80) of claim 14 wherein said color field 
clock (86) operates at a time interval less than the 25 
inverse of the number of said first color components 

in said first color component set to reduce color lag 
as said predetermined shutter speed increases. 

22. The system (80) of claim 14 wherein said first color 30 
components are B, Y and R. 

23. The system (80) of claim 22 wherein said array field 
selector (92) stores B, Y and R first color compo- 
nents and differentiates B - Y to form a U second 35 
color component and differentiates R - Y to form a 

V color component. 

24. The system (80) of claim 22 wherein said array field 
selector (92) stores B, *Y and R first color compo- 40 
nents and temporally differentiates B - Y to form a U 
second color component and temporally differenti- 
ates R - Y to form a V color component. 

45 



50 



55 
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(54) Method and system for field sequential color image capture 



(57) A method of field sequential color image cap- 
ture includes optically capturing a scene; filtering the 
scene through an active color filter (84) to product first 
color components, thereby modifying the spectral trans- 
mission of the scene as a function of time; detecting the 
scene with a single, monochrome sensor (90) and di- 
viding the scene into multiple second color components, 
whereby each first color component of the scene is de- 
tected at a different point in time; aligning the second 
color components in time for each frame interval; storing 
each color component in a memory unit; combining the 
stored second color components into a frame image; 



and processing the frame image for color reproduction 
and format. A system for capturing a field sequential 
color image, includes an optical capture mechanism for 
capturing a scene frame-by-frame in frame intervals; an 
active color filer (84) for producing multiple first color 
components; a single monochrome area sensor(90); an 
array field selector (92) for dividing the scene into mul- 
tiple second color components; multiple memory loca- 
tions for storing each color component; a field-to-frame 
combiner (118) for combining the stored second color 
components into a frame image; and a color reproduc- 
tion processor (120) for processing the frame image for 
color reproduction and format. 
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